The SOLAR System for Sharp Web Archiving

Web archives preserve the history of digital information on the Internet. They are a great asset for media and business analysts and also for experts on intellectual property (e.g, patent offices, IP lawyers) and Internet legislators (e.g. consumer services) to prove or disprove certain allegations. To fulfill this purpose, archives should not only periodically crawl a Web site’s pages but should also assure that the captured pages are a precise representation of the Web site as of a single timepoint. This is a hard problem, since the politeness etiquette and completeness requirement of archive crawlers mandate slow, long-duration crawling while the Web site is changing. This paper presents the SOLAR (Scheduling of Downloads for Archiving of Web Sites) system for sharp Web archiving. SOLAR crawls all pages of a Web site and then re-crawls the visited pages forming visit-revisit intervals. If all visit-revisit intervals overlap and no page changed between its visit and revisit then all pages are “sharp” and captured as if the entire site were downloaded instantaneously. SOLAR judiciously schedules visits and revisits to maximize the number of sharp pages based on the predictions of page-specific change rates. Experiments with synthetic date show SOLAR outperforms existing techniques and captures the sites as sharp as possible.